Methods and apparatus to synchronize second screen content with audio/video programming using closed captioning data

ABSTRACT

A method for synchronized utilization of an electronic device is provided. The method receives closed captioning data from an audio/video content receiver for a set of audio/video content; retrieves detail for a first event occurring in the set of audio/video content, wherein the first event is indicated by the received closed captioning data; and presents content to a user, using the electronic device, based on the retrieved detail.

TECHNICAL FIELD

Embodiments of the subject matter described herein relate generally tothe utilization of closed captioning data for audio/video programming.More particularly, embodiments of the subject matter relate to utilizingclosed captioning data to synchronize second screen content withaudio/video programming.

BACKGROUND

Increasingly, television viewers are using personal computing devices(e.g., smartphones, tablet computers, laptop computers, smart-watches,etc.) while watching television. Audio/video programming providers(e.g., broadcasters, television providers, on-demand programmingproviders, internet programming providers, and the like) may utilizethis opportunity to connect with a viewer via their personal computingdevices. The personal computing device used in conjunction withtelevision viewing is known as a “second screen” or “companion screen”.

Second screen content provides additional information to a userregarding specific audio/video content, to be presented to the userwhile viewing the specific audio/video content. More specifically,second screen content associated with a particular episode of atelevision program is intended to be viewed by a user simultaneouslywith the particular episode of the television program. Second screencontent may include, additional information, interactive content, andcontent related to a specific set of audio/video programming. Morespecific examples may include, without limitation: webpages, pop-uptext, targeted advertising, interactive content, or the like.Additionally, second screen software applications (“apps”) weredeveloped as a way to allow people to become more engaged with aparticular television program during viewing.

Accordingly, it is desirable to provide users with intelligentlyselected content associated with second screen apps, such as presentingcontent directly associated to particular events currently occurring inprogramming while it is being viewed. Furthermore, other desirablefeatures and characteristics will become apparent from the subsequentdetailed description and the appended claims, taken in conjunction withthe accompanying drawings and the foregoing technical field andbackground.

BRIEF SUMMARY

A method for synchronized utilization of an electronic device isprovided. The method receives closed captioning data from an audio/videocontent receiver for a set of audio/video content; retrieves detail fora first event occurring in the set of audio/video content, wherein thefirst event is indicated by the received closed captioning data; andpresents content to a user, using the electronic device, based on theretrieved detail.

A system for processing audio/video content is provided. The systemincludes an audio/video content receiver, configured to provideaudio/video programming to a display device and to provide closedcaptioning data; a second screen electronic device, configured for usein conjunction with viewing the audio/video programming on the displaydevice, wherein the second screen electronic device is configured to:receive the closed captioning data provided by the audio/video contentreceiver; access a remote server to retrieve detail for an eventoccurring in the audio/video programming, wherein the event is indicatedby the closed captioning data; and present content to a user, based onthe retrieved detail.

A non-transitory, computer-readable medium comprising instructionswhich, when executed by a computer, perform a method, is provided. Themethod receives, at a second screen electronic device, closed captioningdata and metadata for a set of recorded audio/video programming duringplayback; identifies a programming event from the received closedcaptioning data; obtains associated information for the programmingevent; and presents supplemental content to a user on the second screenelectronic device, based on the obtained associated information, whereinthe content is presented when the event occurs in the set of audio/videoprogramming.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of the subject matter may be derived byreferring to the detailed description and claims when considered inconjunction with the following figures, wherein like reference numbersrefer to similar elements throughout the figures.

FIG. 1 is a schematic representation of a system for synchronizingsecond screen content with audio/video programming using closedcaptioning data, in accordance with an embodiment;

FIG. 2 is a schematic block diagram representation of a playlistgenerator, suitable for use in the systems shown in FIG. 1;

FIG. 3 is a schematic representation of a playlist, in accordance withan embodiment;

FIG. 4 is a schematic block diagram representation of a second screenapparatus suitable for use in the systems shown in FIG. 1;

FIG. 5 is a flowchart that illustrates a first embodiment of a processfor synchronized utilization of a second screen electronic device and aaudio/video content source; and

FIG. 6 is a flowchart that illustrates a second embodiment of a processfor synchronized utilization of a second screen electronic device and aaudio/video content source.

DETAILED DESCRIPTION

The following detailed description is merely illustrative in nature andis not intended to limit the embodiments of the subject matter or theapplication and uses of such embodiments. As used herein, the word“exemplary” means “serving as an example, instance, or illustration.”Any implementation described herein as exemplary is not necessarily tobe construed as preferred or advantageous over other implementations.Furthermore, there is no intention to be bound by any expressed orimplied theory presented in the preceding technical field, background,brief summary or the following detailed description.

The subject matter presented herein relates to methods and apparatusused by a second screen electronic device to synchronize thesupplemental content it provides to a set of recorded audio/videoprogramming during playback. This synchronization is accomplished usingclosed captioning data received from an audio/video content receiver. Incertain embodiments, a playlist is retrieved from a remote server, whichreferences events, closed captioning reference strings, and appropriatecontent for display on the second screen electronic device when eachevent occurs in the audio/video content during playback. In someembodiments, the system performs an internet search of specific termsoccurring in the closed captioning data, and provides the resultantcontent to a user via the second screen software electronic device.

Referring now to the drawings, FIG. 1 is a schematic representation of asynchronization system 100 that is suitably configured to support thetechniques for synchronizing second screen content with audio/videoprogramming using closed captioning data, described in more detailherein. The synchronization system 100 may include, without limitation:a audio/video content source 102; a audio/video content receiver 104; apresentation device 106; a playlist generator 108; a data communicationnetwork 110; and a second screen electronic device 112. It should beappreciated that FIG. 1 depicts a simplified embodiment of thesynchronization system 100, and that a realistic and practicalimplementation of the synchronization system 100 may include additionalelements or components.

The audio/video content source 102 is suitably configured to provide astream of audio/video content 114 (e.g., broadcast televisionprogramming, on-demand programming, internet programming, or the like)to an audio/video content receiver 104 and to a playlist generator 108.The audio/video content source 102 may utilize any data communicationmethodology, including, without limitation: satellite-based datadelivery, cable-based data delivery, cellular-based data delivery,web-based delivery, or a combination thereof. In this regard, thesynchronization system 100 may include or utilize an audio/video contentdelivery system (not shown). The specific details of such deliverysystems and related data communication protocols will not be describedhere.

During typical operation, the audio/video content receiver 104 receivesaudio/video content 114 (such as primary program content interspersedwith events of interest which may be associated with second screencontent) signaling information, and/or other data via fiber, internet,wireless, or cellular networks, and/or off-air, satellite, or cablebroadcasts. In certain embodiments, the audio/video content receiver 104includes digital video disc (DVD) player technology, Blu-Ray playertechnology, or other video player technology, and receives audio/videocontent via input media. Such input media may also include universalserial bus (USB) storage, external hard disk storage, or the like. Theaudio/video content receiver 104 then demodulates, descrambles,decompresses, and/or otherwise processes the received digital data, andthen converts the received data to suitably formatted video signals thatcan be rendered for viewing, and/or stored for future viewing, by thecustomer on the presentation device 106. The audio/video contentreceiver 104 is further configured to record received video audio/videocontent, and may comprise Digital Video Recorder (DVR) technology. Thus,the audio/video content receiver 104 can record audio/video content 114and, during playback of the recorded content, be synchronized andutilized cooperatively with the second screen electronic device 112 toprovide additional, supplemental content associated with the recordedprogramming.

The audio/video content receiver 104 produces output that iscommunicated to a presentation device 106. Each audio/video contentreceiver 104 may include or cooperate with a suitably configuredpresentation device 106. The presentation device 106 may be implementedas, without limitation: a television set; a monitor; a computer display;a portable electronic device; or any suitable customer appliance withcompatible display capabilities. In various embodiments, eachaudio/video content receiver 104 is a conventional set-top box (STB)commonly used with satellite or cable television distribution systems.In other embodiments, however, the functionality of an audio/videocontent receiver 104 may be commonly housed within a presentationdevice. In still other embodiments, an audio/video content receiver 104is a portable device that may be transportable with or without thepresentation device 106. An audio/video content receiver 104 may also besuitably configured to support broadcast television reception, videogame playing, personal video recording, video player technology (e.g.,DVD, Blu-Ray, etc.), and/or other features as desired. In certain otherembodiments, the audio/video content receiver 104, the presentationdevice 106, and the second screen electronic device 112 may beimplemented as a single computing device, in which the audio/videoprogramming presented by the audio/video content receiver 104 andpresentation device 106 and the second screen content presented by thesecond screen electronic device 112 are presented using a shared screenof the single computing device.

The audio/video content receiver 104 is further configured tocommunicate output data to a second screen electronic device 112,including closed captioning data and timing data for audio/videocontent, via a wireless local area network (WLAN) or any suitablewireless technology. In some embodiments, the audio/video content 114includes a sequence of video frames with associated timing information,formatted in accordance with the Motion Picture Experts Group (MPEG)standard. The closed captioning data identifies the set of audio/videocontent and presents subtitles, or captions, to viewers of the set ofaudio/video content. Closed captioning data is typically used as atranscription of the audio portion of a set of audio/video content, asit occurs (either verbatim or in edited form), sometimes includingdescriptions of non-speech elements. Timing information for each videoframe may comprise a Presentation Time Stamp (PTS) value. A PTS is areference timing value that is generally included in packet mediastreams (digital audio, video or data), according to the MPEG standard.PTS values are used to control the presentation time alignment of suchmedia, through synchronization of separate components within a videocontent stream (e.g., video, audio, subtitles, etc.). In otherembodiments, timing information for each video frame may comprise aProgram Clock Reference (PCR) value. As used in association withcompressed digital video, a PCR value consists of a time stamp thatindicates an associated System Time Clock (STC) value at the time apacket leaves an encoder. Alternatively, an accurate Time of Day clockmay be used.

The second screen electronic device 112 is suitably configured toreceive data from the audio/video content receiver 104 and one or moreremote servers, and to display supplemental content (i.e., “secondscreen content” or “companion screen content”) indicated by the receiveddata. In certain embodiments, the second screen electronic device 112displays this coordinated, supplemental content to a user while the useris viewing recorded audio/video content played back by the audio/videocontent receiver 104. In some embodiments, the second screen electronicdevice 112 displays this coordinated, supplemental content to a userwhile the user is viewing live content, or content during a firstbroadcast or other form of audio/video content distribution, via theaudio/video content receiver 104. The second screen electronic device112 may be implemented as a handheld electronic device, such as a tabletcomputer or smartphone; a laptop computer; a netbook; a smart-watch; orany other type of computing device. In certain embodiments, the secondscreen electronic device 112 may be implemented as the same apparatus asthe audio/video content receiver 104 and the presentation device 106. Insome embodiments, the second screen electronic device 112 may use aspecialized software application (“app”) to perform the functionalityassociated with cooperatively providing second screen content for a userto view while watching the audio/video content provided by theaudio/video content receiver 104.

In certain embodiments, the second screen electronic device 112communicates with one or more remote servers to retrieve the secondscreen content via the data communication network 110. However, in someembodiments, the second screen electronic device 112 receives secondscreen content from the audio/video content receiver 104 itself. In thisexample, the second screen content may be received by the audio/videocontent receiver 104 simultaneously with the audio/video content 114,when the audio/video content 114 is transmitted or “streamed” by anaudio/video content source 102. In other embodiments, the second screencontent may be received by the audio/video content receiver 104 beforeor after receipt of the audio/video content 114. Alternatively, inembodiments where the audio/video content is received at the audio/videocontent receiver 104 via stored digital media (e.g., DVD, Blu-Ray, USB,external hard disk, etc.) the second screen content may be stored on thedigital media, for retrieval by the audio/video content receiver 104during playback. Second screen content provides additional informationto a user regarding specific audio/video content, to be presented to theuser while viewing the specific audio/video content. More specifically,second screen content associated with a particular episode of atelevision program is intended to be viewed by a user simultaneouslywith the particular episode of the television program. Second screencontent may include, without limitation: webpages, pop-up text, targetedadvertising, interactive content, or the like.

In practice, the data communication network 110 may be any digital orother communications network capable of transmitting messages or databetween devices, systems, or components. In certain embodiments, thedata communication network 110 includes a packet switched network thatfacilitates packet-based data communication, addressing, and datarouting. The packet switched network could be, for example, a wide areanetwork, the Internet, or the like. In various embodiments, the datacommunication network 110 includes any number of public or private dataconnections, links or network connections supporting any number ofcommunications protocols. The data communication network 110 may includethe Internet, for example, or any other network based upon TCP/IP orother conventional protocols. In various embodiments, the datacommunication network 110 could also incorporate a wireless and/or wiredtelephone network, such as a cellular communications network forcommunicating with mobile phones, personal digital assistants, and/orthe like. The data communication network 110 may also incorporate anysort of wireless or wired local and/or personal area networks, such asone or more IEEE 802.3, IEEE 802.16, and/or IEEE 802.11 networks, and/ornetworks that implement a short range (e.g., Bluetooth) protocol. Forthe sake of brevity, conventional techniques related to video/mediacommunication systems, video/media broadcasting systems, datatransmission, signaling, network control, and other functional aspectsof the systems (and the individual operating components of the systems)may not be described in detail herein.

The audio/video content source 102 provides media content (such asaudio/video programming, advertising content, music programming, andother video or audiovisual content) to the audio/video content receiver104 and the playlist generator 108. The playlist generator, in turn,provides output in the form of a playlist 116, to a remote server via adata communication network 110. The playlist generator 108 operates inparallel with the normal video delivery system.

The playlist generator 108 is configured to create a playlist of “eventsof interest” occurring in a set of audio/video content that aresynchronized with appropriate second screen content displayed on thesecond screen electronic device 112. Generally, the events may includeanything occurring in the audio/video content which may initiatesimultaneous viewing of associated second screen content. The playlistgenerator 108 is further configured to associate the located events withcorresponding closed captioning data occurring simultaneously in theaudio/video content 114, and to create a playlist 116 to include locatedevents, the associated closed captioning data, and references to secondscreen content associated with each event.

Synchronization occurs when the appropriate, coordinating second screencontent appears at the same time that an associated event occurs duringplayback of the audio/video content. Such supplemental content enhancesthe viewing experience by providing supplemental information and/orinteractive content. For example, a character in a television showplaces a laptop computer on a table and utters the word “Hello!” at thesame time, during playback of the recorded television show. Here, theaction of setting the laptop computer on the table is the event ofinterest, and the word “Hello!” would be the associated closedcaptioning data. The second screen electronic device receives closedcaptioning data (e.g., “Hello!”) indicating the laptop was placed on thetable, and presents appropriate second screen content to the usersimultaneously. Appropriate second screen content, in this example, mayinclude an advertisement for the particular brand of laptop, a webpagefor purchasing the particular brand of laptop, etc.

The playlist generator 108 may be implemented as one or morecomputer-based devices, components, or systems. The playlist generator108 receives audio/video content 114 from the audio/video content source102 for processing, and processes the audio/video content 114 toidentify and/or extract information regarding the stream of audio/videocontent 114. For example, the playlist generator 108 analyzes the streamof audio/video content 114 to locate a plurality of events occurringwithin the audio/video content 114. The playlist generator 108associates each located event with closed captioning data appearingsimultaneously with each event.

The playlist generator 108 may be manually operated, using human input,or may be operated using specialized computer hardware, software, orother technology to determine, designate, and/or record the occurrenceof specific events and to associate the specific events with closedcaptioning data appearing nearest in time to each event (also called aclosed captioning reference string). The playlist generator 108 isfurther configured to associate each of the events with references toappropriate supplemental content for presentation on a second screen.The playlist generator 108 catalogs the events, associated closedcaptioning reference strings, and associated supplemental content as aplaylist.

The playlist generator 108 produces output, comprising the playlist 116for the audio/video content 114, and communicates this playlist 116 toone or more remote servers, via the data communication network 110, forlocalized use by second screen electronic devices 112. In certainembodiments, the playlist 116 is provided outside of the synchronizationsystem 100, and is already located on a remote server for retrieval by asecond screen electronic device 112 via the data communication network110. In this case, the playlist generator 108 is not necessary and istherefore not included in the synchronization system 100.

In a first implementation of the synchronization system 100, the secondscreen electronic device 112 retrieves a playlist 116 from a remoteserver, via the data communication network 110, to indicate appropriatesecond screen content and the closed caption intended to initiate thedisplay of the second screen content. The second screen electronicdevice 112 then receives closed captioning data from the audio/videocontent receiver as the audio/video content 114 is being viewed (e.g.,during playback of the recorded content), and displays appropriatecontent at each event indicated by the playlist 116. In addition, thesecond screen electronic device 112 also receives metadata from theaudio/video content receiver 104 during playback of the audio/videocontent 114. Such metadata is associated with a particular set ofaudio/video content 114, and may include, without limitation: a title, aunique program identifier, a genre, and a description.

In a second implementation of the synchronization system 100, the secondscreen electronic device 112 receives closed captioning data from theaudio/video content receiver 104, locates specific words within theclosed captioning data, and displays second screen content associatedwith the words displayed. To locate the second screen content fordisplay, the second screen electronic device 112 performs a lookup atone or more remote servers to retrieve additional data associated withthe located words.

For simplicity and ease of illustration, FIG. 1 and this descriptionassume that only one playlist generator 108 is deployed in thesynchronization system 100, and that only one audio/video content source102 communicates with the playlist generator 108. It should beunderstood that an embodiment of the synchronization system 100 mayinclude or support any number of playlist generators 108 and any numberof audio/video content sources 102. The audio/video content source 102is implemented as a physically distinct and remote device or systemrelative to the playlist generator 108. Moreover, a physically distinctaudio/video content source 102 may communicate with the playlistgenerator 108 directly or via the data communication network 110 if sodesired.

FIG. 2 is a schematic block diagram representation of a playlistgenerator 200 suitable for use in the systems shown in FIG. 1. Theplaylist generator 200 can be realized as a computer-based component orsystem. The illustrated embodiment of the playlist generator 200generally includes, without limitation: a processor architecture 202; amemory element 204; a user interface 206; a communication module 208; anaudio/video content analysis module 210; and an output generator 212.These components and elements may be coupled together as needed forpurposes of interaction and communication using, for example, anappropriate interconnect arrangement or architecture. It should beappreciated that the playlist generator 200 represents a “full featured”embodiment that supports various features described herein. In practice,an implementation of the playlist generator 200 need not support all ofthe enhanced features described here and, therefore, one or more of theelements depicted in FIG. 2 may be omitted from a practical embodiment.Moreover, a practical implementation of the playlist generator 200 willinclude additional elements and features that support conventionalfunctions and operations.

The processor architecture 202 may be implemented or performed with oneor more general purpose processors, a content addressable memory, adigital signal processor, an application specific integrated circuit, afield programmable gate array, any suitable programmable logic device,discrete gate or transistor logic, discrete hardware components, or anycombination designed to perform the functions described here. Inparticular, the processor architecture 202 may be realized as one ormore microprocessors, controllers, microcontrollers, or state machines.Moreover, the processor architecture 202 may be implemented as acombination of computing devices, e.g., a combination of digital signalprocessors and microprocessors, a plurality of microprocessors, one ormore microprocessors in conjunction with a digital signal processorcore, or any other such configuration.

The memory element 204 may be realized using any number of devices,components, or modules, as appropriate to the embodiment. Moreover, theplaylist generator 200 could include a memory element 204 integratedtherein and/or a memory element 204 operatively coupled thereto, asappropriate to the particular embodiment. In practice, the memoryelement 204 could be realized as RAM memory, flash memory, EPROM memory,EEPROM memory, registers, a hard disk, a removable disk, or any otherform of storage medium known in the art. In certain embodiments, thememory element 204 includes a hard disk, which may also be used tosupport functions of the playlist generator 200. The memory element 204can be coupled to the processor architecture 202 such that the processorarchitecture 202 can read information from, and write information to,the memory element 204. In the alternative, the memory element 204 maybe integral to the processor architecture 202. As an example, theprocessor architecture 202 and the memory element 204 may reside in asuitably designed ASIC.

The memory element 204 can be used to store and maintain information foruse by the playlist generator 200. For example, the memory element 204may be used to store references to selected audio/video content events,along with associated closed captioning data and time offsets. Thememory element 204 may also be utilized to store data concerningapplicable second screen content for each referenced event. Of course,the memory element 204 may also be used to store additional data asneeded to support the operation of the playlist generator 200.

The user interface 206 may include or cooperate with various features toallow a user to interact with the playlist generator 200. Accordingly,the user interface 206 may include various human-to-machine interfaces,e.g., a keypad, keys, a keyboard, buttons, switches, knobs, a touchpad,a joystick, a pointing device, a virtual writing tablet, a touch screen,a microphone, or any device, component, or function that enables theuser to select options, input information, or otherwise control theoperation of the playlist generator 200. For example, the user interface206 could be manipulated by an operator to mark the selected eventsoccurring throughout the audio/video content, as described above.

The communication module 208 is suitably configured to receive andperform processing on signals received by the playlist generator 200 andto transmit signals from the playlist generator 200. The communicationmodule 208 is used to communicate data between the playlist generator200 and one or more remote servers. As described in more detail below,data received by the communication module 208 includes audio/videocontent from an audio/video content source (see, for example, FIG. 1).Data provided by the communication module 208 includes a playlistdetailing events and their associated closed captioning data, timeoffsets, and referenced second screen content for display at the timethe event is viewed by a user. The communication module 208 may leverageconventional design concepts that need not be described in detail here.

The audio/video content analysis module 210 is suitably configured toperform analysis of a set of audio/video content as a whole and/or eachindividual frame contained within a set of audio/video content. In someembodiments, this analysis is performed to determine events occurringwithin a stream of audio/video content 114, as depicted in FIG. 1. Insome embodiments, the analysis may be performed to determine theexistence and location of events of interest occurring throughout thespecific set of audio/video content. In some embodiments, the eventsinclude actions taken, words or phrases uttered, the appearance ordisappearance of actors, scenery, etc. onscreen, and/or anything elsethat occurs in the audio/video content.

The audio/video content analysis module 210 is further configured toassociate the located events with corresponding closed captioning dataoccurring simultaneously in the set of audio/video content. For example,when the set of audio/video content comprises a television show, andwhen an event comprises an action performed by an actor on thetelevision show, the audio/video content analysis module 210 locates theoccurrence of the action, determines the closed captioning dataappearing onscreen at the time the action is performed, and associatesthe performance of the action with the appropriate closed captioningdata.

In certain embodiments, the event occurs at a time that does not line upexactly with closed captioning data. In this case, the playlistgenerator 108 associates the event with closed captioning data appearingnearest in time to the event, and also associates a time offset toindicate the duration of time between the event and the appearance ofthe closed captioning data. Timing information associated with eachindividual frame within the stream of audio/video content 114 may beused by the playlist generator 108 to determine a time offset, and thetiming information may include Presentation Time Stamp (PTS) values,Program Clock Reference (PCR) values, or the like.

The output generator 212 is suitably configured to receive and formatdata obtained by the audio/video content analysis module 210. In someembodiments, the data is formatted into a playlist suitable fortransmission to one or more remote servers for storage and retrieval bya second screen apparatus. The playlist, therefore, conveys theaudio/video content event data in an appropriate format that can bereceived and processed by the second screen apparatus. The outputgenerator 212 creates the playlist to include located events, theassociated closed captioning data, and a time offset (when applicable).

In addition to formatting the data obtained by the audio/video contentanalysis module 210 into a playlist, the output generator 212 alsoincorporates into the playlist a reference to appropriate second screencontent for each event. The output generator 212 receives informationregarding appropriate second screen content for association with eachevent from an external source. Generally, this information is receivedvia the communication module 208, when the playlist generator 200 isoperated using an appropriate computing apparatus, and/or the userinterface 206, when the playlist generator 200 is operated manually.

FIG. 3 is a schematic representation of a playlist 300, in accordancewith an embodiment. As shown, a playlist 300 generally includes a fieldcontaining a program identifier 302. Generally, the program identifier302 acts as a unique identifier for all programming which may use thesame title, and allows for the creation of a single playlist for eachidentified subset of audio/video content.

Each playlist 300 further includes a list of events 304 and, for eachevent 304, a closed captioning reference string 306, a time offset 308,and second screen content detail 302. Each of the events 304 is listedby name and/or description. The closed captioning reference string 306includes the closed captioning data that appears onscreen when the event304 occurs in a subset of audio/video content. Each closed captioningreference string 306 is pre-selected based on its appearance only oncein a particular set of audio/video content. Once the closed captioningreference string 306 appears onscreen during viewing of the subset ofaudio/video content, the closed captioning reference string 306initiates action (e.g., the presentation of additional content) on asecond screen apparatus. The time offset 308 represents the time betweenthe appearance of the closed captioning reference string 306 and theappearance of the referenced event 304 onscreen, as described withregard to FIG. 1.

The second screen content detail 310 includes a reference to, orindication of, specific content for display on a second screen apparatussimultaneously with the occurrence of each of the events. Appropriatesecond screen content may include any form of additional and/orassociated information regarding the set of audio/video content. Incertain embodiments, a reference to appropriate second screen contentmay include a uniform resource locator (URL) for a website, when thereferenced website displays additional data for the associated event,targeted advertising associated with the event, and/or interactivecontent for use while viewing the event. In some embodiments, areference to appropriate second screen content may include arepresentation of text and/or graphics for display while viewing theevent.

FIG. 4 is a schematic block diagram representation of a second screenelectronic device 400 suitable for use in the systems shown in FIG. 1.The second screen electronic device 400 can be realized as anycomputer-based device which may be used in conjunction with anaudio/video content receiver and presentation device (shown as 104 and106, respectively, in FIG. 1). The illustrated embodiment of a secondscreen electronic device 400 generally includes a processor architecture402; a memory element 404; a user interface 406; a wirelesscommunication module 408; an audio/video content analysis module 410;and a second screen content module 412. These elements and features of asecond screen electronic device 400 may be operatively associated withone another, coupled to one another, or otherwise configured tocooperate with one another as needed to support the desiredfunctionality—in particular, displaying second screen content indicatedby received closed captioning data, as described herein. For ease ofillustration and clarity, the various physical, electrical, and logicalcouplings and interconnections for these elements and features are notdepicted in FIG. 4. Moreover, it should be appreciated that embodimentsof the second screen electronic device 400 will include other elements,modules, and features that cooperate to support the desiredfunctionality. For simplicity, FIG. 4 only depicts certain elements thatrelate to the second screen content display techniques described in moredetail herein.

The processor architecture 402, the memory element 404, and the userinterface 406, are similar in configuration and function to theircounterpart items described above in the context of the playlistgenerator 200. Accordingly, common features and operations of theseelements of the second screen electronic device 400 will not beredundantly described here. However, in some embodiments, the userinterface 406 for the second screen electronic device 400 furthercomprises a display for viewing second screen content, which may includeinteractive content, targeted advertising, pop-up text, and/oradditional data related to simultaneously-playing audio/video content.

The wireless communication module 408 is suitably configured tocommunicate data between the second screen electronic device 400 and oneor more remote servers and/or one or more devices within a wirelessrange of the second screen electronic device 400. Generally, thewireless communication module 408 transmits and receives signalscompatible with a wireless local area network (WLAN) (e.g., using anIEEE 802.11 standard). The wireless communication module 408 may utilizean an ad-hoc network, a Bluetooth network, a PAN, or a WLAN, incombination with the Internet, a cellular network, a broadband network,a wide area network, or the like.

As described in more detail below, data received by the wirelesscommunication module 408 may include, without limitation: closedcaptioning data and/or metadata associated with audio/video contentduring playback, received from a audio/video content receiver; aplaylist from a remote server; and second screen content associated withaudio/video content, for display during playback of the associatedaudio/video content. Data provided by the wireless communication module408 may include, without limitation: requests to remote servers toretrieve a playlist and/or indicated second screen content.

The audio/video content analysis module 410 is configured to analyzeclosed captioning data and timing data received by the wirelesscommunication module 408 during playback of audio/video content on anaudio/video content receiver. The audio/video content receiver is inwireless communication range of the second screen electronic device 400,and the closed captioning data and the timing data are transmittedwirelessly by the audio/video content receiver, during playback of therecorded content. In practice, the audio/video content analysis module410 may be implemented with (or cooperate with) the processorarchitecture 402 to perform at least some of the functions andoperations described in more detail herein. In this regard, theaudio/video content analysis module 410 may be realized as suitablywritten processing logic, application program code, or the like.

In a first implementation, the audio/video content analysis module 410identifies the audio/video content being played back and retrieves aplaylist for the identified audio/video content (such as the playlist300 illustrated in FIG. 3) from a remote server, via the wirelesscommunication module 408. The audio/video content analysis module 410performs a lookup to compare the received closed captioning data to theclosed captioning reference strings associated with events cataloged inthe playlist. When a match between a received caption and a closedcaptioning reference string in the playlist is found, the audio/videocontent analysis module 410 reports this result, and an indication ofappropriate second screen content in the playlist, to the second screencontent module 412 for display. Closed captioning data is continuouslyreceived during playback of the recorded audio/video content, and theaudio/video content analysis module 410 continuously performs lookups tomake comparisons and locate matching data in the playlist.

In some embodiments, a timing offset is also presented in the playlist,and the second screen electronic device 400 uses both the closedcaptioning data and the timing data received at the wirelesscommunication module 408 to present second screen content at theappropriate time. For example, suppose an event includes an onscreencharacter placing a suitcase down on the floor. This event may beassociated with a closed captioning reference string including the word“suitcase”, and a timing offset of three seconds. In this example, theaudio/video content analysis module 410 may receive the closedcaptioning data, including the word “suitcase”, perform the lookup todetermine a match in the playlist for the event, determine that there isa three-second timing offset associated with the event in the playlist,and communicate this information to the second screen content module 412for further use.

In a second implementation, the audio/video content analysis module 410does not use a playlist to determine when events of interest areoccurring onscreen, but instead parses closed captioning data, receivedduring playback of recorded content, to locate caption words indicativeof events occurring onscreen. The audio/video content analysis module410 then communicates these words to the second screen content module412 for appropriate retrieval and presentation of second screen contentassociated with the words. Generally, closed caption words used includeproper nouns. Within the context of this application, a “proper noun”may be defined as a capitalized noun in the transmitted closedcaptioning data, to include names of persons, places, or things ofsignificant relevance to the current set of audio/video content.

In certain embodiments, the second screen electronic device 400 mayutilize a “black list” of unauthorized terms, to prevent unwantedcaption words from being transmitted to the second screen content module412 to be used for the retrieval and presentation of second screencontent. For example, words designated by the Federal CommunicationsCommission (FCC) as being unfit for broadcast television during daytimehours may be present on the “black list” of words, to prevent thepresentation of content associated with profane language. The “blacklist” of words may be user-configurable, and may include not onlyprofanity, but any word for which a user does not wish to viewadditional content. In this case, if a user does not enjoy a particulartelevision show or appreciate recent headlines associated with aparticular actor, the user simply adds the name of the television showor actor to the “black list”, and the second screen content module 412will not obtain or present supplemental content based on these words.

In some embodiments, the second screen electronic device 400 may utilizea “white list” of words. Such a “white list” may be used to activelyseek particular words within the closed captioning content, and toretrieve and present second screen content associated with those words.A “white list” of words may be used to ensure that particular topics,people, places, or things of interest to a user will be activelyutilized for the presentation of second screen content, even whenconventional methods of selecting words within received closedcaptioning data would have overlooked them.

In certain embodiments, the audio/video content analysis module 410locates the caption words in the transmitted closed captioning data, anddetermines a context for each of the caption words in order to presentsecond screen content that is relevant to the audio/video content. Thecontext may be determined by analyzing received closed captioning datausing, without limitation, one of the following methodologies: the typeof audio/video content and the caption word; or the caption word itself,along with surrounding group of caption words.

The second screen content module 412 is configured to retrieve anddisplay appropriate second screen content to supplement and accompany aset of audio/video content during playback on an audio/video contentreceiver (such as reference 104 in FIG. 1). Generally, an audio/videocontent receiver transmits closed captioning data and timing data to thesecond screen electronic device 400, where it is analyzed by theaudio/video content analysis module 410 to identify events of interestin the set of audio/video content using closed captioning data (based ona pre-defined playlist or identified caption words appearing in theclosed captioning data). The second screen content module 412 utilizesthe information regarding these events of interest, and displaysappropriate supplemental content on the second screen electronic device400.

Supplemental content presented on a second screen may include, withoutlimitation: graphics, text, a website address (such as a URL),advertisements, interactive user content (e.g., surveys, social mediainteraction, etc.) or other relevant content appropriate for display ona second screen electronic device 400. In the first implementation,where a playlist is used, the appropriate second screen content isindicated by the playlist (shown as reference 310 in FIG. 3). In thesecond implementation, where caption words from the closed captioningdata are identified, the second screen content module 412 initiates aninternet search (via the wireless communication module 408) to retrievecontent relevant to a caption word. Such internet searches are performedin accordance with a determined context of the caption word, and mayyield links to “official” websites, news/informational websites,websites featuring interactive content, and the like.

FIG. 5 is a flowchart that illustrates a first embodiment of a process500 for synchronized utilization of a second screen electronic deviceand a audio/video content source. In this particular embodiment, aplaylist is retrieved and used to supply supplemental content on asecond screen electronic device. The various tasks performed inconnection with process 500 may be performed by software, hardware,firmware, or any combination thereof. For illustrative purposes, thefollowing description of process 500 may refer to elements mentionedabove in connection with FIGS. 1-4. In practice, portions of process 500may be performed by different elements of the described system, e.g., aplaylist generator, an audio/video content receiver, or a second screenelectronic device. It should be appreciated that process 500 may includeany number of additional or alternative tasks, the tasks shown in FIG. 5need not be performed in the illustrated order, and process 500 may beincorporated into a more comprehensive procedure or process havingadditional functionality not described in detail herein. Moreover, oneor more of the tasks shown in FIG. 5 could be omitted from an embodimentof the process 500 as long as the intended overall functionality remainsintact.

For clarity and ease of illustration, it is assumed that the process 500begins by receiving closed captioning data from an audio/video contentreceiver for a set of audio/video content (step 502). Generally, closedcaptioning data is transmitted, via wireless communication network, to asecond screen electronic device, where the process 500 is beingexecuted. The closed captioning data identifies the set of audio/videocontent and presents subtitles, or captions, to viewers of the set ofaudio/video content. Closed captioning data is typically used as atranscription of the audio portion of a program as it occurs (eitherverbatim or in edited form), sometimes including descriptions ofnon-speech elements. In certain embodiments, metadata associated withthe set of audio/video content is also transmitted. Metadata may includea programming identifier, a title, description, genre, or otherinformation relevant to the particular set of audio/video content.

Next, the process 500 retrieves a playlist comprising a list of eventsoccurring in the set of audio/video content, and detail associated witheach of the list of events (step 504). A playlist is created and storedfor future use, and this concept is beyond the scope of the presentpatent application. The playlist may be stored at a remote server or onportable media (e.g., DVD, Blu-Ray, USB storage, etc.), and may beretrieved for use. In some embodiments, the playlist is receivedconcurrently with programming, such as that included in a cable orsatellite broadcast. An exemplary embodiment of a playlist is depictedin FIG. 3. The events cataloged in the playlist are occurrences in theset of audio/video content that have some significance—they are pointsin the program at which supplemental content will be displayed via asecond screen electronic device. The detail associated with each eventincludes a closed captioning reference string, or in other words, aclosed caption that appears at the point in time that the event occursin the set of audio/video content. Detail associated with each eventalso includes an indication of appropriate second screen content fordisplay at the time the event occurs. In certain embodiments, theparticular playlist retrieved for a particular set of audio/videocontent is indicated by the received metadata. For example, metadataincluding a particular program identifier may initiate retrieval of aplaylist including the same program identifier.

In certain embodiments, a closed captioning reference string does notappear at the exact time that the event of interest occurs in the set ofaudio/video content during playback. In this case, a time offset is alsoincluded as part of the playlist, and the received timing data is usedto determine the appropriate time for second screen content to bedisplayed. For example, a closed caption indicating an event may appeartwo seconds prior to the occurrence of the event. Here, a time offset oftwo seconds would be included in the playlist for the event.

The process 500 then presents content to a user, based on the playlist,using the second screen electronic device (step 506). During playback ofthe set of audio/video content, the process 500 continuously receivesclosed captioning data (step 502), and compares the received closedcaptioning data to the closed captioning reference strings cataloged inthe retrieved playlist. When a received caption matches a closedcaptioning reference string for an event on the playlist, the process500 identifies a reference to, or an indicator of, appropriate secondscreen content (also cataloged in the playlist), retrieves theappropriate second screen content from system memory and/or a remoteserver, and displays the second screen content on the second screenelectronic device.

FIG. 6 is a flowchart that illustrates a second embodiment of a process600 for synchronized utilization of a second screen electronic deviceand a audio/video content source. In this particular embodiment, captionwords from the closed captioning data of a set of audio/video contentare identified and used to supply supplemental content on a secondscreen electronic device. First, the process 600 receives closedcaptioning data from an audio/video content receiver for the set ofaudio/video content. The process 600 then identifies a word from thereceived closed captioning data (step 604). In certain embodiments, theword is a proper noun, or in other words, a capitalized name identifyinga person, place, or thing of significance that is discussed, appearsand/or occurs within the set of audio/video content. In someembodiments, the word may be selected based on a frequency of use withina program.

In certain exemplary embodiments, after identifying the caption word(step 602), based on the occurrence of the caption word in the closedcaptioning data of a set of audio/video content during playback, theprocess 600 compares the caption word to a predefined “black list” ofunauthorized terms. When the caption word (e.g., proper noun) appears onthe predefined list of unauthorized terms, the process 600 terminatesaction for the caption word and presents no content based on the captionword or associated with the particular event.

Next, the process determines the context of the identified caption word(step 606). The context may be determined by analyzing a genre of theset of audio/video content in combination with the word, or by analyzingthe identified word and a group of surrounding words. For example, theset of audio/video content may include a broadcast televisionpresentation, which may be identified by a genre or type. Genres ofaudio/video programming may include one of the following, withoutlimitation: news, sports, music, comedy, drama, movie, or the like. Whendetermining a context for the identified word using the genre ofaudio/video content and the word itself, the process 600 first locates acaption word in the received closed captioning data, and then retrievesthe indicated genre from received metadata.

Once a caption word has been identified from the received closedcaptioning data, and the genre of show has been determined, theaudio/video content analysis module 410 performs a lookup to determinethe appropriate context. For example, if the identified caption word isthe proper noun “Boston”, and the type of show has been determined to benews, a determined context indicates that “Boston” is referring to thecity of Boston. In another example, if the identified caption word isthe proper noun “Boston” and the type of show is music, then the contextwould indicate that “Boston” is referring to a music band named Boston.In a final example, if the identified caption word is the proper noun“Boston” and the type of show is sports, then the determined contextwould indicate that “Boston” is referring to a sports team, such as theBoston Celtics.

In addition, once a caption word has been identified, the audio/videocontent analysis module 410 may simply present choices to the user andallow the user to determine the context for the caption word at issue.For example, if the caption word “Boston” has been identified as arelevant proper noun, the process 600 may perform an internet search andpresent user-selectable options for context of the use of the captionword “Boston” in the set of audio/video content currently in playback.In this case, choices may include content relevant to the city ofBoston, the band named Boston, sports teams located in Boston (e.g.,Celtics, Red Sox, etc.), and the like. Selections indicating “band”,“basketball team”, “baseball team”, and/or “city” may be presented tothe user, and the user may input their choice for presentation ofrelevant second screen content.

When determining a context for the identified caption word using theword itself and a group of surrounding caption words, the process 600may utilize an algorithm to perform an evaluation of the group ofcaption words. For example, the proper noun “Ford” may indicate a brandof truck or a former President of the United States. In some embodimentsof such an algorithm, the process 600 may search the closed captioningdata for the words “Ford” and “F150” or the words “Ford” and “truck” inthe same sentence, and if the condition is satisfied, then the contextof the word “Ford” is determined to be relating to a Ford truck. Inother embodiments, the process 600 may search for the same wordsdescribed above, but within twenty (20) words of each other. When thiscondition is satisfied, the context of the word “Ford” is againdetermined to be relating to a Ford truck.

After determining the context of the identified word (step 606), theprocess 600 accesses a remote server to retrieve detail for an event, orseries of events, occurring in the set of audio/video content (step608). The “event” is the occurrence of something onscreen, whether it issomething spoken, an action taken, the entrance or exit of an actor, theintroduction of a change in discussion topic, and the like. Occurrenceof the event is indicated by the closed captioning data, or morespecifically, the identified caption word (e.g., proper noun) in itsappropriate context. Generally, the retrieval of detail is in the formof an internet search for the proper noun. In certain embodiments,detail may include additional relevant information, advertisements,interactive content, or the like.

Once appropriate detail has been retrieved for the identified captionword (step 608), the process 600 presents content to a user, using asecond screen electronic device, based on the retrieved detail and thedetermined context of the identified word (step 610). The process 600retrieves detail associated with the identified word by performing aninternet search, and the retrieved detail includes internet searchresults. In certain embodiments, the internet search results arenarrowed, using the determined context for the identified word. Oncereduced internet search results are obtained (and possibly narrowedusing the context), results are presented to the user in the form ofsecond screen content on a second screen electronic device. Results maybe presented in various ways, including executing a command to open aweb browser to present the following, without limitation: a first resultis presented to the user in the form of a webpage; a designated numberof the first results are presented as tabs in a web browser; or a resultmarked “official” may be presented to a user as a webpage. In someembodiments, user-selectable options may be presented to the user, fromwhich the user may choose a result for display on the second screenelectronic device.

Techniques and technologies may be described herein in terms offunctional and/or logical block components, and with reference tosymbolic representations of operations, processing tasks, and functionsthat may be performed by various computing components or devices. Suchoperations, tasks, and functions are sometimes referred to as beingcomputer-executed, computerized, software-implemented, orcomputer-implemented. In practice, one or more processor devices cancarry out the described operations, tasks, and functions by manipulatingelectrical signals representing data bits at memory locations in thesystem memory, as well as other processing of signals. The memorylocations where data bits are maintained are physical locations thathave particular electrical, magnetic, optical, or organic propertiescorresponding to the data bits. It should be appreciated that thevarious block components shown in the figures may be realized by anynumber of hardware, software, and/or firmware components configured toperform the specified functions. For example, an embodiment of a systemor a component may employ various integrated circuit components, e.g.,memory elements, digital signal processing elements, logic elements,look-up tables, or the like, which may carry out a variety of functionsunder the control of one or more microprocessors or other controldevices.

When implemented in software or firmware, various elements of thesystems described herein are essentially the code segments orinstructions that perform the various tasks. The program or codesegments can be stored in a computer-readable or processor-readablemedium or transmitted by a computer data signal embodied in a carrierwave over a transmission medium or communication path. The“processor-readable medium”, “machine-readable medium”, or“computer-readable medium” may include any medium that can store ortransfer information. Examples of the processor-readable medium includean electronic circuit, a semiconductor memory device, a ROM, a flashmemory, an erasable ROM (EROM), a floppy diskette, a CD-ROM, an opticaldisk, a hard disk, a fiber optic medium, a radio frequency (RF) link, orthe like. The computer data signal may include any signal that canpropagate over a transmission medium such as electronic networkchannels, optical fibers, air, electromagnetic paths, or RF links. Thecode segments may be downloaded via computer networks such as theInternet, an intranet, a LAN, or the like.

Some of the functional units described in this specification have beenreferred to as “modules” in order to more particularly emphasize theirimplementation independence. For example, functionality referred toherein as a module may be implemented wholly, or partially, in softwarefor execution by various types of processors. An identified module ofexecutable code may, for instance, comprise one or more physical orlogical modules of computer instructions that may, for instance, beorganized as an object, procedure, or function. Nevertheless, theexecutables of an identified module need not be physically locatedtogether, but may comprise disparate instructions stored in differentlocations that, when joined logically together, comprise the module andachieve the stated purpose for the module.

Indeed, a module of executable code may be a single instruction, or manyinstructions, and may even be distributed over several different codesegments, among different programs, and across several memory devices.Similarly, operational data may be embodied in any suitable form andorganized within any suitable type of data structure. The operationaldata may be collected as a single data set, or may be distributed overdifferent locations including over different storage devices, and mayexist, at least partially, merely as electronic signals on a system ornetwork.

While at least one exemplary embodiment has been presented in theforegoing detailed description, it should be appreciated that a vastnumber of variations exist. It should also be appreciated that theexemplary embodiment or embodiments described herein are not intended tolimit the scope, applicability, or configuration of the claimed subjectmatter in any way. Rather, the foregoing detailed description willprovide those skilled in the art with a convenient road map forimplementing the described embodiment or embodiments. It should beunderstood that various changes can be made in the function andarrangement of elements without departing from the scope defined by theclaims, which includes known equivalents and foreseeable equivalents atthe time of filing this patent application.

What is claimed is:
 1. A method for synchronized utilization of anelectronic device, the method comprising: receiving closed captioningdata from an audio/video content receiver for a set of audio/videocontent; retrieving detail for a first event occurring in the set ofaudio/video content, wherein the first event is indicated by thereceived closed captioning data, by: accessing a remote server toretrieve a playlist comprising a list of events occurring in the set ofaudio/video content and detail associated with each of the list ofevents; wherein the detail associated with each of the events comprisesa closed captioning reference string, a time offset, and a reference tosecond screen content for display after occurrence of the closedcaptioning reference string and passage of the time offset in the set ofaudio/video content; performing a lookup to compare the closedcaptioning data to a plurality of closed captioning reference stringscataloged in the playlist, to locate a match, wherein the plurality ofclosed captioning reference strings comprises the closed captioningreference string; and when a match is located, identifying appropriatesecond screen content cataloged in the playlist; and presenting theappropriate second screen content to a user, during occurrence of thefirst event during playback of the set of audio/video content, by theelectronic device, based on the retrieved detail, wherein theappropriate second screen content comprises the second screen content.2. The method of claim 1, wherein the closed captioning data comprises aproper noun, and wherein the first event is associated with occurrenceof the proper noun in the closed captioning data of the set ofaudio/video content.
 3. The method of claim 2, further comprising:identifying the proper noun, based on the occurrence of the proper noun;determining a genre for the set of audio/video content; determining acontext for the proper noun based on the determined genre of the set ofaudio/video content; and presenting the content to the user based on theretrieved detail and the determined context.
 4. The method of claim 2,further comprising: identifying the proper noun, based on the occurrenceof the proper noun; evaluating a group of words surrounding the propernoun; determining a context for the proper noun based on the evaluatedgroup of words; and presenting the content to the user based on theretrieved detail and the determined context.
 5. The method of claim 1,wherein the presenting step further comprises: executing a command toopen a web browser, via the electronic device; and presenting a websiteto the user, using the opened web browser, wherein the retrieved detailcomprises a uniform resource locator (URL), and wherein the contentcomprises the website.
 6. The method of claim 1, wherein the presentingstep further comprises: presenting a plurality of selectable options tothe user, via the electronic device, wherein the detail comprises theplurality of selectable options; and receiving user input to select oneof the plurality of selectable options; wherein the content comprisesthe selected one of the plurality of selectable options.
 7. A system forprocessing audio/video content, the system comprising: an audio/videocontent receiver, configured to provide audio/video programming to adisplay device and to provide closed captioning data; a second screenelectronic device, configured for use in conjunction with viewing theaudio/video programming on the display device, wherein the second screenelectronic device is configured to: receive the closed captioning dataprovided by the audio/video content receiver; access a remote server toretrieve detail for an event occurring in the audio/video programming,by: accessing the remote server to retrieve a playlist comprising a listof events occurring in the set of audio/video content and detailassociated with each of the list of events; wherein the detail for theevent comprises a closed captioning reference string, a time offset, anda reference to second screen content for display after occurrence of theclosed captioning reference string and passage of the time offset in theset of audio/video content; and wherein the event is indicated by theclosed captioning data; perform a lookup to compare the closedcaptioning data to a plurality of closed captioning reference stringscataloged in the playlist, to locate a match, wherein the plurality ofclosed captioning reference strings comprises the closed captioningreference string; and when a match is located, identify appropriatesecond screen content cataloged in the playlist; and present theappropriate second screen content to a user, during occurrence of thefirst event during playback of the set of audio/video content, based onthe retrieved detail, wherein the appropriate second screen contentcomprises the second screen content.
 8. The system of claim 7, whereinthe audio/video content receiver comprises a set-top box (STB)configured to communicate with the second screen electronic device overa wireless local area network (WLAN).
 9. The system of claim 7, furthercomprising a single computing device, wherein the single computingdevice comprises the audio/video content receiver and the second screenelectronic device; and wherein the audio/video programming and thecontent are presented using a shared screen of the single computingdevice.
 10. The system of claim 7, further comprising: a playlistgenerator, configured to: manually designate the event occurring in theaudio/video programming; and associate the event with a closedcaptioning reference string, wherein the closed captioning referencestring comprises a subset of the closed captioning data; and a playlistcreation module, configured to: further associate the manuallydesignated event with the content for presentation via the second screenelectronic device; and generate the playlist cataloging the event, theclosed captioning reference string, and the content; wherein the detailcomprises the playlist.
 11. The system of claim 7, wherein the retrieveddetail comprises at least the event, an associated closed captioningreference string, and associated content detail.
 12. The system of claim11, wherein the content comprises a website, and wherein the associatedcontent detail comprises a uniform resource locator (URL).
 13. Thesystem of claim 11, wherein the content comprises text displayed via thesecond screen electronic device, and wherein the associated contentdetail comprises a representation of the text.